Various technologies exist for capturing and reconstructing three-dimensional data (3D) of an object or environment (e.g., a building, or a room within a building). The input image data of these existing technologies may come from a two-dimensional (2D) image sensor or a 3D sensor that captures depth information as well as 2D imagery (pixel color, texture, etc.), such as a time-of-flight camera, a structured light camera, or a stereo camera. The 3D reconstruction process often involves finding a position and orientation of the 2D camera when it captured various images.
The 3D reconstructions made by these existing processes are often imperfect. For example, existing 3D modeling processes can have difficulty handling certain types of objects, such as thin objects, highly reflective objects, partially or completely transparent objects, objects with highly complex geometry, or very distant objects. Additionally, the 3D models generated by these existing processes may not have the same level of texture resolution as the raw 2D imagery associated therewith, for example due to hardware limitations.
Accordingly, rather than viewing an imperfect view of a 3D reconstruction of an object or environment, it is desirable to be able to view the 2D images or 2D panoramas of the object or environment that were used to generate the 3D reconstruction, if available. More particularly, a smooth interface is desired for switching from interactive free exploration of the 3D model to viewing desired 2D imagery and back again. Such a smooth interface has been previously generated using 2D and 3D image data captured from sophisticated cameras fixed to a tripod and configured to rotate 360 degrees relative to the object or environment at various fixed points and angles. These cameras can generate structured panoramic imagery at a specific capture point. The smooth interface combines a 3D mesh of the object or environment along with the several 360 degree photographic 2D panoramas of the object or environment.
Mobile devices (e.g., tablets, smartphones, wearable devices, etc.) are becoming configured with 2D and 3D sensors capable of capturing color and depth data. Thus, mobile devices are becoming capable of capturing image data of a real world environment that can be used to generate a 3D model of the environment. However, for mobile devices, the method of image data capture is significantly different than that of the sophisticated cameras attached to a rotating tripod. For example, when capturing image data of an environment using a handheld mobile device, it is impractical for a user to capture a series of structured 360 panoramas of the environment. Rather, the captured image data is more free form as the mobile capture device moves about or around the environment, capturing images from various non-uniform positions and angles. Further the field of view offered by a mobile capture device is significantly less than that of sophisticated cameras configured to capture structured panoramas. Accordingly, the techniques used to generate a smooth interface that facilitates viewing and navigating a 3D model using 2D and 3D image data captured from cameras configured to generate several structured 360 degree photographic 2D panoramas are insufficient when applied to image data captured using a mobile device.